{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How SequenceToSequence works"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\">\n",
    "<ul class=\"toc-item\">\n",
    "<li><span><a href=\"#Introduction\" data-toc-modified-id=\"Introduction-1\">Introduction</a></span></li>\n",
    "<li><span><a href=\"#Prerequisites\" data-toc-modified-id=\"Prerequisites-2\">Prerequisites</a></span></li>\n",
    "<li><span><a href=\"#Data-preparation\" data-toc-modified-id=\"Data-preparation-3\">Data preparation</a></span></li>\n",
    "<li><span><a href=\"#SequenceToSequence model\" data-toc-modified-id=\"SequenceToSequence-model-4\">SequnceToSequence model</a></span></li>\n",
    "<ul class=\"toc-item\">\n",
    "<li><span><a href=\"#Model-instantiation\" data-toc-modified-id=\"Model-instantiation-4.1\">Model instantiation</a></span>\n",
    "<li><span><a href=\"#How-to-choose-an-appropriate-backbone-for-your-dataset?\" data-toc-modified-id=\"How-to-choose-an-appropriate-backbone-for-your-dataset?-4.2\">How to choose an appropriate backbone for your dataset?</a></span>    \n",
    "<li><span><a href=\"#Model-training\" data-toc-modified-id=\"Model-training-4.3\">Model training</a></span>\n",
    "    <ul class=\"toc-item\">\n",
    "        <li><span><a href=\"#Finding-optimum-learning-rate\" data-toc-modified-id=\"Finding-optimum-learning-rate-4.3.1\">Finding optimum learning rate</a></span>\n",
    "        <li><span><a href=\"#Evaluate-model-performance\" data-toc-modified-id=\"Evaluate-model-performance-4.3.2\">Evaluate model performance</a></span>\n",
    "        <li><span><a href=\"#Validate-results\" data-toc-modified-id=\"Validate-results-4.3.3\">Validate results</a></span></li> \n",
    "    </ul>\n",
    "</ul>\n",
    "<li><span><a href=\"#Model-inference\" data-toc-modified-id=\"Model-inference-5\">Model inference</a></span></li>\n",
    "<li><span><a href=\"#References\" data-toc-modified-id=\"References-6\">References</a></span></li>\n",
    "</ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`SequenceToSequence` model can be used for multiple tasks that fall under the category of sequence translation. \n",
    "Sequence translation is the task of translating an input sequence to an output sequence of any length (independent of the input sequence length).  \n",
    "Few such tasks are:\n",
    "\n",
    "*\t[Machine translation.](https://en.wikipedia.org/wiki/Machine_translation#:~:text=Machine%20translation%2C%20sometimes%20referred%20to,from%20one%20language%20to%20another.)\n",
    "*\t[Summarization.](https://en.wikipedia.org/wiki/Automatic_summarization)\n",
    "*\t[Question answering](https://en.wikipedia.org/wiki/Question_answering), etc.\n",
    "\n",
    "<img src=''>\n",
    "<center>Figure1: High level view of sequence translation</a></center>\n",
    "\n",
    "`SeqenceToSequence` model in `arcgis.learn.text` module is built on top of [Hugginface transformers](https://huggingface.co/transformers/) library. This library provides access to a wide range of transformer architectures. \n",
    "\n",
    "The transformer architecture as proposed in the [Attention is all you need](https://arxiv.org/abs/1706.03762) paper, consists of an encoder block and a decoder block. Many of the latest transformer-based architectures like BERT, RoBERTa, ALBERT utilizes only the encoder part of the transformer. Whereas other models like GPT, GPT-2, etc. utilizes only the decoder part of the architecture. But, for SequenceToSequence  tasks we need both the encoder and decoder.  T5, Bart, MBart are few examples of such architectures, which preserve both the encoder and decoder.\n",
    "\n",
    "For a detailed walkthrough of transformer architecture, refer to Jay Almmar's blogpost [[1]](#References).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Refer to the section [Install deep learning dependencies of arcgis.learn module](https://developers.arcgis.com/python/guide/install-and-set-up/#Install-deep-learning-dependencies) for detailed explanation about deep learning dependencies.\n",
    "* Labeled data: For SequenceToSequence to learn, it needs to see examples that have been translated in a way that the model is expected to translate an input text into. Head to the Data preparation section to see the supported formats for training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `SequenceToSequence` class in `arcgis.learn.text` module can consume labeled training data in [CSV](https://en.wikipedia.org/wiki/Comma-separated_values) file format\n",
    "\n",
    "\n",
    "Sample input data format for `SeqenceToSequence` model training:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=''>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training data must have two columns, one for input text and the other for translated output text. In the above example, `non_std_address` is the input text column, which has addresses from the U.S. in a non-standard format. `std_address` is the output text column which has the input address translated into a standard format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Data preparation** involves splitting the data into training and validation sets, creating the necessary data structures for loading data into the model and so on. The `prepare_textdata` function can directly read the training samples in the above specified format and automate the entire process. While calling this function, the user has to provide the following arguments:\n",
    "- **path**&nbsp;&ensp;&emsp;&emsp;&emsp;&emsp;- &emsp;&emsp;&emsp;&emsp;        The **full directory path** where the **training file** is present\n",
    "- **task**&nbsp;&ensp;&emsp;&nbsp;&emsp;&emsp;&emsp;- &emsp;&emsp;&emsp;&emsp;       The **task** for which the **dataset** is being prepared, for `SequenceToSequence` model  \n",
    " &nbsp;&ensp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;pass **\"sequence_translation\"** as the task name.\n",
    "- **train_file**&nbsp;&ensp;&emsp;&nbsp;&ensp;&nbsp;-  &emsp;&emsp;&emsp;&emsp; The file name containing the **training data**. Supported file format/extension is **.csv**\n",
    "- **text_columns**&nbsp;&nbsp;-  &emsp;&emsp;&emsp;&emsp; The column name in the csv file that will be used as **input feature**.\n",
    "- **label_columns** -  &emsp;&emsp;&emsp;&emsp;The column denoting the translated text.\n",
    "\n",
    "Some pre-processing functions are also provided like removing [HTML tags](https://html.com/tags/) from the text or removing the [URLs](https://en.wikipedia.org/wiki/URL) from the text. Users can decide if these pre-processing steps are required for their dataset or not.\n",
    "\n",
    "**A note on the dataset**\n",
    "- The data is collected around 2020-04-30 by [OpenAddresses](http://openaddresses.io).\n",
    "- The data licenses can be found in `data/address_standardization_correction_data/LICENSE.txt`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from arcgis.learn import prepare_textdata\n",
    "from arcgis.learn.text import SequenceToSequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = prepare_textdata(path='data/', batch_size=16, task='sequence_translation', \n",
    "                        text_columns='non_std_address', label_columns='std_address', \n",
    "                        train_file='address_standardization.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`show_batch()` method can be used to visualize the training samples, along with labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_f5d58cfa_3ada_11eb_848a_f40270a5e290 th {\n",
       "          text-align: left;\n",
       "    }    #T_f5d58cfa_3ada_11eb_848a_f40270a5e290row0_col0 {\n",
       "            text-align:  left;\n",
       "        }    #T_f5d58cfa_3ada_11eb_848a_f40270a5e290row0_col1 {\n",
       "            text-align:  left;\n",
       "        }    #T_f5d58cfa_3ada_11eb_848a_f40270a5e290row1_col0 {\n",
       "            text-align:  left;\n",
       "        }    #T_f5d58cfa_3ada_11eb_848a_f40270a5e290row1_col1 {\n",
       "            text-align:  left;\n",
       "        }    #T_f5d58cfa_3ada_11eb_848a_f40270a5e290row2_col0 {\n",
       "            text-align:  left;\n",
       "        }    #T_f5d58cfa_3ada_11eb_848a_f40270a5e290row2_col1 {\n",
       "            text-align:  left;\n",
       "        }    #T_f5d58cfa_3ada_11eb_848a_f40270a5e290row3_col0 {\n",
       "            text-align:  left;\n",
       "        }    #T_f5d58cfa_3ada_11eb_848a_f40270a5e290row3_col1 {\n",
       "            text-align:  left;\n",
       "        }    #T_f5d58cfa_3ada_11eb_848a_f40270a5e290row4_col0 {\n",
       "            text-align:  left;\n",
       "        }    #T_f5d58cfa_3ada_11eb_848a_f40270a5e290row4_col1 {\n",
       "            text-align:  left;\n",
       "        }</style><table id=\"T_f5d58cfa_3ada_11eb_848a_f40270a5e290\" ><thead>    <tr>        <th class=\"col_heading level0 col0\" >non_std_address</th>        <th class=\"col_heading level0 col1\" >std_address</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                                <td id=\"T_f5d58cfa_3ada_11eb_848a_f40270a5e290row0_col0\" class=\"data row0 col0\" >366, richland avenue, athens, ohio, 45701.0, us</td>\n",
       "                        <td id=\"T_f5d58cfa_3ada_11eb_848a_f40270a5e290row0_col1\" class=\"data row0 col1\" >366, richland ave, athens, oh, 45701.0, us</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                                <td id=\"T_f5d58cfa_3ada_11eb_848a_f40270a5e290row1_col0\" class=\"data row1 col0\" >524, parnell drive, branson, missouri, 65616, us</td>\n",
       "                        <td id=\"T_f5d58cfa_3ada_11eb_848a_f40270a5e290row1_col1\" class=\"data row1 col1\" >524, parnell dr, branson, mo, 65616, us</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                                <td id=\"T_f5d58cfa_3ada_11eb_848a_f40270a5e290row2_col0\" class=\"data row2 col0\" >26645.0, freedom valley dr, washburn, wi, 54891.0, us</td>\n",
       "                        <td id=\"T_f5d58cfa_3ada_11eb_848a_f40270a5e290row2_col1\" class=\"data row2 col1\" >26645.0, freedom valley dr, washburn, wi, 54891.0, us</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                                <td id=\"T_f5d58cfa_3ada_11eb_848a_f40270a5e290row3_col0\" class=\"data row3 col0\" >15728.0, 430th avenue, delavan, minnesota, 56023.0, us</td>\n",
       "                        <td id=\"T_f5d58cfa_3ada_11eb_848a_f40270a5e290row3_col1\" class=\"data row3 col1\" >15728.0, 430th ave, delavan, mn, 56023.0, us</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                                <td id=\"T_f5d58cfa_3ada_11eb_848a_f40270a5e290row4_col0\" class=\"data row4 col0\" >7129, tr 664, nan, ohio, 44624, us</td>\n",
       "                        <td id=\"T_f5d58cfa_3ada_11eb_848a_f40270a5e290row4_col1\" class=\"data row4 col1\" >7129, tr 664, nan, oh, 44624, us</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7fa560203b38>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SequenceToSequence model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `SequenceToSequence` model training and inferencing workflows are similar to computer vision models in `arcgis.learn`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model instantiation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "SequenceToSequence class constructor accepts two named arguments `data`(required) and `backbone`(optional). \n",
    "\n",
    "`data`: `TextDataObject` object prepared with the `prepare_textdata()` function.\n",
    "\n",
    "`backbone`: A pretrained transformer model based on the transformer architecture of choice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to choose an appropriate backbone for your dataset?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`supported_backbones` attribute can be called on `SequenceToSequence` class to see the supported backbones (transformer architectures) by the model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['T5', 'Bart', 'Marian']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SequenceToSequence.supported_backbones"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`available_backbone_models()` method accepts one of the `supported_backbones` and returns a list (this list is not exhaustive) of pretrained models for that backbone/architecture."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Users can choose a supported backbone and a pretrained model based on the task the model is to be trained on. It is preferable to choose a model that was trained on a similar task and data.\n",
    "\n",
    "For instance, if a user is training a model for summarizing English text. It would be most appropriate to choose `T5` architecture and `t5-base-finetuned-summarize-news` model, as this model was trained for summarization downstream task.\n",
    "\n",
    "Visit [HuggingFace model zoo](https://huggingface.co/models) and filter the tags based on the task to find all the available pretrained models for that particular task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['t5-small',\n",
       " 't5-base',\n",
       " 't5-large',\n",
       " 't5-3b',\n",
       " 't5-11b',\n",
       " 'See all T5 models at https://huggingface.co/models?filter=t5 ']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SequenceToSequence.available_backbone_models('T5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = SequenceToSequence(data,backbone='t5-base')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Finding optimum learning rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In machine learning, the `learning rate`[[2]](#References) is a tuning parameter that determines the step size at each iteration while moving toward a minimum of a loss function, it represents the speed at which a machine learning model \"learns\"\n",
    "\n",
    "- If the **learning rate is low**, then model training will take a lot of time because steps towards the minimum of the loss function are tiny.\n",
    "- If the **learning rate is high**, then training may not converge or even diverge. Weight changes can be so big that the optimizer overshoots the minimum and makes the loss worse.\n",
    "\n",
    "We have to find an **optimum learning rate** for the dataset we wish to train our model on. To do so we will call the `lr_find()` method of the model.\n",
    "\n",
    "**Note**\n",
    "\n",
    "- A user is not required to call the `lr_find()` method separately. If `lr` argument is not provided while calling the `fit()` method then `lr_find()` method is internally called by the `fit()` method to find the optimal learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3dd3xUZdr/8c+VTkihJKGFlgAiIjV0FMuq6CpWdu2FJuquu6s+utV1dfd5XJ/fqouuXRHsBVyxLWtZQaQmQOi9JaGlAAmBhJTr98cMPjEmIYE5c2Yy1/v1mldmzrln5pvJZK65z7nPfURVMcYYE7rC3A5gjDHGXVYIjDEmxFkhMMaYEGeFwBhjQpwVAmOMCXFWCIwxJsQ5XghEJFxEVojIx3WsixaRd0Rki4gsEZFuTucxxhjzff7oEfwCWF/PuonAAVXtATwB/NUPeYwxxtTgaCEQkVTgx8BL9TS5HJjhvf4+cL6IiJOZjDHGfF+Ew4//JHA/EF/P+k5ADoCqVorIIaAtUFDfAyYlJWm3bt18HNMYY5q3rKysAlVNrmudY4VARC4F9qtqloicU1+zOpb9YM4LEZkCTAHo0qULmZmZPstpjDGhQER21rfOyU1Do4BxIrIDeBs4T0Rer9UmF+gMICIRQCJQVPuBVPUFVc1Q1Yzk5DoLmjHGmJPkWCFQ1d+oaqqqdgOuBb5S1RtrNZsD3OK9fo23jc2CZ4wxfuT0PoIfEJGHgUxVnQO8DLwmIlvw9ASu9XceY4wJdX4pBKr6NfC19/qDNZaXAeP9kcEYY0zd7MhiY4wJcVYIjDEmxFkhMMaYEGeFwBhjApyq8vcvNrNud7Ejj+/3UUPGGGMar7paeeijtcxctJOjFVX06Zjg8+ewQmCMMQGqsqqaX89ezftZuUw+qzsPjD3NkeexQmCMMQHoWGU1v3xnBZ+u3suvftSLu8/vgVNzclohMMaYAPTQR2v5dPVefv/j05l0Vpqjz2U7i40xJsDMXp7Lm0t2MXVMuuNFAKwQGGNMQNmwt5jffrCa4WltuO/CXn55TisExhgTIIrLKrjj9eUkxEQy7bqBRIT75yPa9hEYY0wAqKiq5p53VrKr6AhvTR5OSnyM357begTGGOOyiqpq7n5rBV+s389Dl/VhaPc2fn1+KwTGGOOi40XgszV7efDSPtw0opvfM1ghMMYYl9QuAhNGd3clhxUCY4xxyZ8/Xsdna/byBxeLAFghMMYYV7y5ZBczFu1k8lndmehiEQArBMYY43dLthXy4IdrGNMrmV9ffLrbcawQGGOMP+UUHeGON5bTpW0s064bSHiYM/MHNYUVAmOM8ZOdhaXc8NISKqqqeenmDBJbRLodCbADyowxxi/W5B3i1unLqKquZuaEoaQlx7kd6TtWCIwxxmGLthYyeWYmCTERzJwykh4pgVMEwAqBMcY4KnNHEbdMX0rXNrHMnDiUDokt3I70A1YIjDHGIVvzDzNpZiaprVrwzu0jaNMyyu1IdbKdxcYY44D8knJunb6UiDDh1duGBmwRAOsRGGOMzx05VsnEGcsoKDnG21OG06VtrNuRGmQ9AmOM8bFHPl7HmrxDPH39QPp3buV2nBOyQmCMMT60YtcB3l6Ww4RR3Tn/9HZux2kUxwqBiMSIyFIRyRaRtSLypzra3Coi+SKy0nuZ5FQeY4xxWlW18uCHa0mJj+aXF/jnNJO+4OQ+gnLgPFU9LCKRwAIR+UxVF9dq946q/szBHMYY4xdvLd3F6rxDTLtuIHHRwbML1rGkqqrAYe/NSO9FnXo+Y4xxU+Hhcv537kZGpLXlsn4d3I7TJI7uIxCRcBFZCewHPlfVJXU0u1pEVonI+yLSuZ7HmSIimSKSmZ+f72RkY4w5KY9+toHS8koevvwMRNyfSK4pHC0EqlqlqgOAVGCoiPSt1eQjoJuq9gO+AGbU8zgvqGqGqmYkJyc7GdkYY5rsw5V5vJeVy+Sz0+jZLt7tOE3ml1FDqnoQ+BoYW2t5oaqWe2++CAz2Rx5jjPGVTftK+PWs1WR0bc09QbSDuCYnRw0li0gr7/UWwI+ADbXa1NyQNg5Y71QeY4zxtZKyCqa+nkXL6Aj+ccMgIsODc0S+k7u1OwAzRCQcT8F5V1U/FpGHgUxVnQPcLSLjgEqgCLjVwTzGGOMzqsr9769iZ+ER3pg0jHYJMW5HOmlOjhpaBQysY/mDNa7/BviNUxmMMcYp72Xl8tmavfzm4t4MT2vrdpxTEpz9GGOMcdHh8kr+d+5GBnZpxZSz09yOc8qsEBhjTBM99/VW8kvK+cOlfYJuqGhdrBAYY0wT5B08yovfbOPyAR0Z1KW123F8wgqBMcY0wV8/8wx+vH9sb5eT+I4VAmOMaaTluw4wJ3s3U85Oo1OrwDvl5MmyQmCMMY306KcbSImPZuqYdLej+JQVAmOMaYSl24tYuqOIO89Jp2UQzSzaGFYIjDGmEZ75egttW0bx0yFd3I7ic1YIjDHmBNbkHeLrjflMGN2dFlHhbsfxOSsExhhzAs/O20p8dAQ3Du/qdhRHWCEwxpgGbMs/zKer93DjiK4ktoh0O44jrBAYY0wDnp+3jajwMCaM6u52FMdYITDGmHrsOXSU2Sty+emQziTHR7sdxzFWCIwxph4vzN+GKkw+K/gnlmuIFQJjjKlD4eFy3lq6i8sHdKJzm1i34zjKCoExxtRh+rc7KK+s5o5zmndvAKwQGGPMDxSXVTBj0Q7GntGeHinBdzL6prJCYIwxtby+eCclZZXceU4Pt6P4RcgWggOlxyivrHI7hjEmwBw9VsXL32xnTK9kzkxNdDuOX4RUIVBVsnYWcecbWQz+8+c86p1X3BhjjnsvK4fC0mPcdW5o9AbAwZPXB5rMHUU88sl6snMOkhATQafWLfj32n082ExONWeMOXWqyhuLd9E/NZGh3du4HcdvQqZHEBYmHDpyjIcvP4NFvzmf289OJ+/gUbbml7odzRgTINbuLmbjvhLGZ3R2O4pfhUyPYFCX1nx17zmEhXm+/Y/plQzA/E359EiJczOaMSZAvJ+VS1REGJf16+h2FL8KmR4B8F0RAOjcJpbuSS2ZvznfxUTGmEBRXlnFP1fmcWGfdiTGNs/J5eoTUoWgtjG9klm8rZCyChs9ZEyo+2r9fg4eqeCawaluR/G7kC4EZ/dKoqyimswdB9yOYoxx2ftZubRLiOasnsluR/G7kC4Ew9PaEhUeZpuHjAlx+0vK+HpTPlcOTCU8LPRGEYZ0IYiNiiCjW2vmbbRCYEwo+3DFbqqqNSQ3C4GDhUBEYkRkqYhki8haEflTHW2iReQdEdkiIktEpJtTeepzdq9kNu4rYe+hMn8/tTEmAKgq72flMrBLq5AdQehkj6AcOE9V+wMDgLEiMrxWm4nAAVXtATwB/NXBPHX6bhipbR4yJiQdP3bgqkGh2RsABwuBehz23oz0XrRWs8uBGd7r7wPni58P8+3dPp6U+Gjmb7JCYEwomr08j6jwMC7r18HtKK5xdB+BiISLyEpgP/C5qi6p1aQTkAOgqpXAIaBtHY8zRUQyRSQzP9+3H9giwlk9k1mwpYDKqmqfPrYxJrBVVFUzJzuP809PoVVslNtxXONoIVDVKlUdAKQCQ0Wkb60mdX37r91rQFVfUNUMVc1ITvb90K4L+rTj4JEKFm4t9PljG2MC1zeb8yk4fCykNwuBn0YNqepB4GtgbK1VuUBnABGJABKBIn9kqumc05KJj45gTvZufz+1McZFs5fn0To28rt9haHKyVFDySLSynu9BfAjoPa8z3OAW7zXrwG+UtUf9AicFhMZzkV92zN3zV47ytiYEHHoaAX/XrePcf07EhUR0iPpHe0RdAD+IyKrgGV49hF8LCIPi8g4b5uXgbYisgW4B/i1g3kaNK5/R0rKK/najikwJiR8tnoPxyqrQ36zEDg4+6iqrgIG1rH8wRrXy4DxTmVoipHpbUmKi2JOdh5j+7Z3O44xxmGzV+SRntySfiFyFrKGhHZ/qIaI8DAuObMDX67fT0lZhdtxjDEOyik6wtLtRVw1KNVOTIUVgu+5fEBHyiur+XzdPrejGGMc9PKC7YQJXDGwk9tRAoIVghoGdWlNp1Yt+HCljR4yprnatK+E1xbv5LqhXejUqoXbcQKCFYIaRITL+ndkwZYCCg+Xux3HGONjqsqfPlpLXHQE9154mttxAoYVglrG9e9IVbXy2uKdbkcxxvjY3LV7+XZLIfde2Is2LUP3SOLarBDU0qdjApf268BTX21h+S47YY0xzUVZRRWPfLye3u3juX5oF7fjBBQrBHX4y5Vn0j4hhl+8vYJiG0FkTLPw/Lxt5B08yoOX9SEi3D76arJXow6JLSKZdt0Adh8s4w//XIMLBzsbY3xoX3EZz83byiVntmdkepLbcQKOFYJ6DO7ahl+c35MPV+7m3cwct+MYY07Bk19sorK6mgfG9nY7SkCyQtCAu87twbDubXhg1momzchky/6S762vrKqmutp6C8YEss37SnhnWQ43Du9K17Yt3Y4TkBybYqI5CA8TXr1tKC8v2MZz87Zx4RPzubSfZ1TR5v0lbC8oJbFFFBf0SeHCM9ozMr0t0RHhbsc2xtTw139toGVUBD8/r6fbUQKWFYITaBEVzs/O68n1w7ry9FdbeGfZLpLjo+mREs95vduRc+AIc1bu5q2lOSTERHDziG5MGN3dhqYZEwCWbCvki/X7uX/safY/2QAJth2hGRkZmpmZ6XaM7ymvrGLhlkLeWZbD3HV7iYkI57qhXZg6Jo2UhBi34xkTklSVK55ZyP7iMv5z3znERIZ2b11EslQ1o6511iPwgeiIcM7tncK5vVPYsr+EZ77eyoxFO3hn2S7uPLcHE0d3r/dNWFR6DMC+rRjjY3PX7iU75yCPXdMv5IvAiViPwCE7Ckr570/X8+91+0ht3YK7z+vJyB5t6dSqBSLCjoJSnp+/lVlZeYjA7WPSuWNMOi2i7A1rzKlSVS6ZtoDyiio+v2cM4WE2w6j1CFzQLaklL9ycwcItBTz88Trun7UKgJT4aLontWTZjiIiwsMYn5FKSVkl077czPuZOdw/tjfD0trQLj6GMHvzGnNSPl+3j/V7ivnb+P5WBBrBegR+UF2trNtTzPJdB1i+8wAb9pYw5rRkJo7q/t0+hKXbi3hozlrW7SkGICoijM6tW3BZ/47cdW4PIu1ISGMaRVW57OkFlJRV8uU9Y+woYq+GegRWCAJIVbWyZFsh2wtL2VV4hHV7ivlmcwEDOrfi79cOaNQY6LyDR1myrZCxfdsTG2UdPhN6vtqwjwmvZvLYNf34SUZnt+MEDCsEQeyj7N389oPVVFcrk89O4+CRCjbuLWFr/mFSEqLpl9qK/qmJHKusZk72bpbt8EyUN7RbG6bfNoSW0VYMTOhQVa74x7cUHTnGV/eeYz3pGqwQBLm8g0f51TsrWbq9iNiocHq1iyc9OY69xUdZlXuIkrJKAHqmxHH5gI4kxkbx0Jy1DOrSium3DSXOioEJEV9v3M+t05fx6FVncq3NMPo9trM4yHVq1YK3Jw+n4HA5SXHR39uJXF2t7CgspVqV9OS4786/2iY2irvfXsEtryzl1duGEB8T6VZ8Y/zmmf9spVOrFlw1KNXtKEGlUf0mEWkpImHe671EZJyI2CeLH4WFCSkJPxxJFBYmpCXH0SMl/nsn4f5xvw48fd1AsnMOcuPLS787XsGY5io75yBLdxQxYXR3oiJsk1BTNPbVmg/EiEgn4EvgNuBVp0IZ37j4zA48e+NgNuwp5prnFpJ38KjbkYxxzMsLthMfHcFPMqw30FSNLQSiqkeAq4CnVPVKoI9zsYyvXNCnHa9NHEZ+STlXP7OQTftKTnwnY4LM7oNH+WT1Hq4d2tk2g56ERhcCERkB3AB84l1m+xeCxNDubXj39hFUq3LNswt5PyvXTrZjmpUZC3egqtwyspvbUYJSYwvBL4HfAB+o6loRSQP+41ws42und0hg1h0j6dkunvvey+aGl5awvaDU7VjGnLLS8kreXLqLi8/sQGrrWLfjBKVGfatX1XnAPADvTuMCVb3byWDG9zq3ieW920fw5tJd/PVfG7joyfmMTG9L+4QY2iXE0Do2kspqpbJaUfX0JAZ1afW9ndDGBJr3MnMoKatk0ujubkcJWo0qBCLyJjAVqAKygEQReVxV/7eB+3QGZgLtgWrgBVX9e6025wAfAtu9i2ar6sNN/SVM44WFCTcO78qFfdrx+OebWLP7EGvyiiksLaeurUVd28ZyxYBOXDM4lc5t7NuWCSxV1cor3+5gUJdWDOzS2u04Qaux2/n7qGqxiNwAfAo8gKcg1FsIgErgXlVdLiLxQJaIfK6q62q1+0ZVL21ycnNKUhJiePTqft/drqiqpqSskvAwITJcqKhU/r1uL/9cmce0rzYz7avNXNinHZPPSmNw19bWSzAB4dstBewqOsJ/XXSa21GCWmMLQaT3uIErgKdVtUJEGtzbqKp7gD3e6yUish7oBNQuBCYARIaHff+cCFEwPqMz4zM6s+fQUV5fvJPXF+9i7tp9nNkpkaHd23Bau3hOa++52Hzvxg2zlueSEBPBBX3auR0lqDW2EDwP7ACygfki0hUobuyTiEg3YCCwpI7VI0QkG9gN3Keqaxv7uMY/OiS24L8u6s1d5/Zg1vI83s/K5Y0lOymrqAYgIkw4vUMCAzq3YnhaWy7u296m0DaOKymrYO7avVw9KNW+iJyik55rSEQiVLWyEe3i8Oxo/ouqzq61LgGoVtXDInIJ8HdV/cEZpkVkCjAFoEuXLoN37tx5UpmN71RVKzlFR9iwt4TVeQdZsesg2TkHKT1WxeCurfmfq86kV7t4t2OaZuzdZTncP2sVs+8cySDbP3BCpzzpnIgkAn8EzvYumgc8rKqHTnC/SOBjYK6qPt6I59kBZKhqQX1tQnHSuWBRVa38c0Uef/5kHYfLK7ljTDp3ntvDvq0ZR/zk+UUUlJTz5b1jbJ9VIzRUCBp7HMErQAnwE++lGJh+gicV4GVgfX1FQETae9shIkO9eQobmckEmPAw4erBqXxxzxgu69eRaV9tYfxzi2xqC+NzOUVHWLq9iKsHp1oR8IHGFoJ0Vf2jqm7zXv4EpJ3gPqOAm4DzRGSl93KJiEwVkaneNtcAa7z7CKYB16od8hr02sZF8/hPB/DizRnsKCjlsqcW8O2Wejt5xjTZrOW5iMAVAzu5HaVZaOzO4qMiMlpVFwCIyCigwa953rYNlmpVfRp4upEZTJC5oE87PvzZKG5/LYubXl7CL3/Ui4mju9vJcswpUVVmL89jRFpbOrVq4XacZqGxPYKpwD9EZId3O/7TwO2OpTLNRlpyHB/cNYqLz+zA459vYsT/fMlj/9rA/pIy2LoV7rwTEhIgLMzz8847PcuNqceyHQfYVXSEq+2cAz7TqEKgqtmq2h/oB/RT1YHAeY4mM81GXHQE/7h+ELPvHMnI9CSenbeV305+jIq+Z6IvvQQlJaDq+fnSS9CvH3z2mduxTYCak51Hi8hwxvZt73aUZqNJfXRVrXnswD3Ak76NY5qzQV1a89xNg8lZtpqU0dcQeazsh40qKjyXa66BVasgPd3/QU3AUlW+WLefs3sl2SZGHzqV0/jYrnpzUjpPf5YorWq4UUUFPPGEfwKZoLEmr5i9xWVc0Md6A750KoXARveYk/P660hFRcNtKirgtdf8k8cEjc/X7yNM4LzeKW5HaVYa7FuJSAl1f+ALYLvrzck5fNi37UzI+GLdPjK6tvn+vFjmlDXYI1DVeFVNqOMSr6q2gc6cnLg437YzISH3wBHW7SnmR32sN+Brp7JpyJiTc+ONENnweWU1MhJuuslPgUww+HL9fgB+dLrNNOprVgiM/9177wkLQbmEc+znv/BTIBMMvli/j/TklqQlW0/R16wQGP9LT4f334fY2B8WhMhIKmNaMHXcr3lgZSk244gBKC6rYPG2Qn5k5x1whBUC446LL/YcJzBlyvePLJ4yhYg1q8m443o+WJHHY3M3up3UBIB5G/OpqFIusM1CjrAdvsY96enw9NOeSy13pSm7D5Xx7NdbaZ8Qwy0ju/k/nwkYX6zfR9uWUXZeYodYITABSUR4eNwZ5JeU89BHa2kbF8Wl/Tq6Hcu4oLKqmv9s2M+FZ7Qn3M585wjbNGQCVkR4GE9dN5CMrq351TsrbSrrELUy5yDFZZV2EJmDrBCYgBYTGc5LNw8hLSmOKTMzWZPX4EnxTDM0b1M+4WHCqB5JbkdptqwQmICXGBvJzIlDaRUbxa3Tl9oZz0LMvE35DOzcisQWDQ85NifPCoEJCu0SYpgxYSjlFdXc/lomR4+dYNI60ywUHi5ndd4hxvRKdjtKs2aFwASNHilx/P26AazdXcwDs1bZMQYhYMGWAlRhzGlWCJxkhcAElfN6t+O+C09jTvZunp+/ze04xmHzNubTpmUUfTsmuh2lWbNCYILOneekc2m/Dvz1Xxv4fN0+t+MYh1RXK/M353N2zyTCbNioo6wQmKAjIjx2TT/6dUrkrjeWM29TvtuRjAPW7Smm4PAx2yzkB1YITFCKjYpg5oRh9EjxDCtduNWOMWhujhf4s3paIXCaFQITtBJjI3lt4lC6to1l4quZLNtR5HYk40PzNubTt1MCSXHRbkdp9qwQmKDWNi6a1ycNo0NiDBNeXcaW/SVuRzI+UFxWQdauAzZs1E+sEJiglxIfw8yJQ4mOCGfCq5kUlR5zO5I5RQu3FFBVrZxtm4X8wgqBaRZSW8fy4s2D2Vdcxu2vZVJeaQecBbNvNhcQFx3BoK4226g/WCEwzcbALq3520/6s2zHAX49a7UdcBbEFmwpYHhaGyLD7SPKH+xVNs3Kpf06cu8FvfhgRR5PfrHZ7TjmJOQUHWFn4RFG2yRzfuNYIRCRziLyHxFZLyJrReQHJ6AVj2kiskVEVonIIKfymNDxs/N6MH5wKn//cjPvLstxO45pom82e4YCj7b9A37j5IlpKoF7VXW5iMQDWSLyuaquq9HmYqCn9zIMeNb705iTJiL891Vnsre4jN98sJp2iTE2+iSIfLulgPYJMaQnt3Q7SshwrEegqntUdbn3egmwHuhUq9nlwEz1WAy0EpEOTmUyoSMyPIxnbhhEr3bx3Pl6lp3HIEhUVSvfbi1gdM8kRGxaCX/xyz4CEekGDASW1FrVCajZd8/lh8XCmJMSHxPJ9FuHkNgikokzlrH3UJnbkcwJrN19iINHKjirp+0f8CfHC4GIxAGzgF+qanHt1XXc5QdDPURkiohkikhmfr7NK2Mar31iDC/fOoTS8iomzlhGaXml25FMA47vHxiZboXAnxwtBCISiacIvKGqs+tokgt0rnE7Fdhdu5GqvqCqGaqakZxs23pN05zeIYGnrh/I+j3F/OLtFVRV27DSQLVgcwG928eTHG/TSviTk6OGBHgZWK+qj9fTbA5ws3f00HDgkKrucSqTCV3nnpbCQ+PO4Iv1+/nLJ+vdjmPqcPRYFVk7D9hmIRc4OWpoFHATsFpEVnqX/RboAqCqzwGfApcAW4AjwG0O5jEh7uYR3diWX8or326nV7s4rh3axe1IpoalO4o4VlVtw0Zd4FghUNUF1L0PoGYbBe5yKoMxtf3+x6ezNf8wf/hwDekpcQzp1sbtSMZrweZ8osLDGGp/E7+zI4tNSIkID+Pp6waR2jqWqa9lkXfwqNuRjNeCLYUM7tqaFlHhbkcJOVYITMhJjI3kxZszOFZZzeQZmRw5ZiOJ3Hag9Bjr9xQzqkdbt6OEJCsEJiT1SIlj2nUDWb+3mHvfzabaRhK5avG2QgBG2LBRV1ghMCHr3N4p/Pbi0/lszV4em7vR7TghbeHWQmKjwumXmuh2lJDk5KghYwLepLO6s72wlOfmbaV7Uiw/HWIjidywcGsBQ7vbtNNusVfdhDQR4U/jzuCsnkn87oM1fLulwO1IIWd/cRlb80sZmW77B9xihcCEvMjwMP5xwyDSkltyx+tZbMs/7HakkLLo+P6BNNs/4BYrBMYACTGRvHzLECLCw5g0M5NDRyvcjhQyFm4pJCEmgj4dE9yOErKsEBjj1blNLM/eMIhdhUe4+y2bk8hfFm4rYHhaW8LDbNppt1ghMKaGYWlteeSKvszblM+jn9mcRE7LKTpCTtFRRtj+AVfZqCFjarluaBc27i3hxW+20z0pjuuH2UgipxzfP2DTTrvLCoExdfj9j09nZ2Epv//nalrHRnLxmXbiPCcs2lpI25ZR9GoX53aUkGabhoypQ0R4GM/cMJiBXVrzi7dX2rBSB6gqC7cWMCK9rZ2W0mVWCIypR4uocF65ZQjdk1oyZWYm2TkH3Y7UrGzNP8y+4nLbPxAArBAY04DE2EhmThxK65ZR3PzKUtbkHXI7UrMxJ3sPInB+73ZuRwl5VgiMOYF2CTG8OWk4cdER3PDSEisGPqCqfLAil1HpSbRPjHE7TsizQmBMI3RpG8vbU6wY+ErmzgPkFB3lyoGd3I5isEJgTKN1bvN/xeD6FxdbMTgFs5fn0SIynLF927sdxWCFwJgmOV4M4mMiuenlJWzaV+J2pKBTVlHFJ6t2c9EZ7WgZbSPYA4EVAmOaqHObWN6YNIzI8DCuf3GJTVLXRP/ZsJ/iskquHJTqdhTjZYXAmJPQLaklb04ehqpyw0tLyCk64nakoDF7RR7J8dGMsmGjAcMKgTEnqUdKPK9NHMaRY1Vc+8JiKwaNcKD0GF9v3M/l/TsSYSehCRj2lzDmFPTpmMAbk4ZxuLzSikEjfLRqNxVVypWDbLRQILFCYMwp6tsp8XvFYFehFYO6HCg9xrQvt9AvNZE+HezcA4HECoExPnC8GJQeq+TaFxaxo6DU7UgB58E5azl09BiPXtXP5hYKMFYIjPGR48WgrLKa8c8vsqGlNXyyag8fZe/m7vN62pnIApAVAmN86IyOibwzZTgC/PT5RXbQGZBfUs7v/7mafqmJ3HFOuttxTB2sEBjjYz3bxfPu7SOIjYrguhcXk7WzyO1IrlFVfvfBakqPVfG38f1tpFCAcuyvIiKviMh+EVlTz/pzROSQiA27LZUAAA3PSURBVKz0Xh50Kosx/tYtqSXvTh1BUlw0N7y0hK827HM7kisem7uRf6/bx/0XnUbPdvFuxzH1cLI8vwqMPUGbb1R1gPfysINZjPG7Tq1a8N7UEfRMiWfyzCxmZeW6HcmvXvpmG89+vZUbhnVh4ujubscxDXCsEKjqfCB0+8TGAElx0bw1ZTjD09pw73vZvDB/q9uR/GJWVi5//mQ9l5zZnocv72ujhAKc2xvsRohItoh8JiJn1NdIRKaISKaIZObn5/sznzGnLC46glduHcKP+3Xgvz/dwCMfr6O6Wt2O5Zgv1u3j/lmrGNWjLU/8dADhYVYEAp2bU/8tB7qq6mERuQT4J9Czroaq+gLwAkBGRkbz/Q8yzVZ0RDhPXTuQ5LhoXl6wnf0l5fy/8f2Ijgh3O5pPLdhcwJ1vLKdvxwSevymj2f1+zZVrPQJVLVbVw97rnwKRIpLkVh5jnBYWJvzxsj78+uLefJS9m1tfWUZxWYXbsXxm2Y4iJs/MJC25JTMmDCXOppgOGq4VAhFpL94NhyIy1Jul0K08xviDiDB1TDqP/6Q/y3YUcc2zC8k9EPxTUqzKPciE6cvokBjDaxOH0So2yu1IpgmcHD76FrAIOE1EckVkoohMFZGp3ibXAGtEJBuYBlyrqrbZx4SEqwalMnPCUPYcKuPKZxayKveg25FOWk7REW6dvozE2EjemDyM5PhotyOZJpJg++zNyMjQzMxMt2MY4xOb95Vw6/RlFJaWM+3agVx4RnCdurG0vJKrn11I3sGjfHjXKNKS49yOZOohIlmqmlHXOrdHDRkT0nq2i+efd43itHbx3P56FjMW7nA7UqNVVyv3vLuSTftKePr6QVYEgpgVAmNclhzvOdbgR6e3449z1vLnIBle+uSXm5m7dh+/veR0xvRKdjuOOQVWCIwJALFRETx342BuHdmNlxZs5643l3P0WJXbser1yao9TPtyM+MHp9pRw82AFQJjAkS4d3jp7398Ov9au5ern10YkGc8W517iHvfW8ngrq3585V21HBzYIXAmAAiIkw6K41Xbh1CzoEjjHt6Ad9uKXA71nf2FZcxaeYy2raM5vmbBtsBY82EFQJjAtC5p6Uw52ejSYqL5qaXl/Di/G24PcKvrKKKKTMzKSmr5KVbMkiKs2GizYUVAmMCVPeklnxw1ygu7NOev3y6nrveXM7h8kpXslRXK/e9l82qvEM8+dMBnG7nHG5WrBAYE8DioiN49sZB/PaS3sxdu49xTy9gswunwHxs7kY+XrWHB8b2DrpjHcyJWSEwJsCJCFPOTueNScMoPlrJZU8v4LVFO/y2qej1xTt5bp7nvAK3n53ml+c0/mWFwJggMTytLZ/cPZqh3dvyhw/Xcsv0Zew9VOboc365fh8PfriG83qn8KdxZ9gIoWbKCoExQaRdQgwzbhvCI1f0Zdn2Ii56cj4fZe925LlW5R7k52+toE/HBJ66bqCdb7gZs7+sMUFGRLhpeFc+/cVZdE9qyc/fWsHdb63g0BHfTWmdU3SECa8uo3VsFK/cMoSWNqV0s2aFwJgg1T2pJe9PHcE9F/Ti09V7uOjJ+czbdOpn8DtQeoxbpi+lokqZMWEIKQkxPkhrApkVAmOCWER4GHef35PZd46kZXQ4t7yylEkzMtleUHpSj1dWUcXkmZnkHjjKizdn0CMl3seJTSCyQmBMM9AvtRWf3H0WD4ztzaKtBVz4xDz+/PE6Cg6XN/oxyiqquOP1LDJ3HuCJnwxgaPc2DiY2gcTOR2BMM7O/pIy/zd3Eu1k5RIaHcc3gVCaflUb3pJb13qe0vJJJMzJZvL2Q/77yTK4b2sWPiY0/NHQ+AisExjRTW/MP89I325i1PI+Kqmr6dEigU6sWdGzVgs5tYundPp7e7eOJCA/jtulLyc49xP8b348rB6a6Hd04wAqBMSEsv6Sc1xfvZFXuQfIOHiXvwFFKa0xxHR0RRrUqT103iLF97ajh5qqhQmBjwoxp5pLjo/nVBb2+u62qFJYeY8OeEjbsLWZrfimX9e/AyPQkF1MaN1khMCbEiAhJcdGM7hnN6J724W9s1JAxxoQ8KwTGGBPirBAYY0yIs0JgjDEhzgqBMcaEOCsExhgT4qwQGGNMiLNCYIwxIS7oppgQkXxgZyObJwKHmri+9rL6btdcXntZElDQyIyNyVrfuoaynuh6oGeta1lTszr5968rq1Ov6almDca/fzBlDZb3aitVTa7z0VW12V6AF5q6vvay+m7XXF57GZDpy6z1rWso64muB3rWepY1KauTf/96XktHXtNTzRqMf/9gyhpM79X6Ls1909BHJ7G+9rL6bn90gmVN1dB961vXUNYTXQ/0rPWtbwon//41rzv9mta33t6rJ2bv1UY8ftBtGgoGIpKp9czyF2gsq+8FS06wrE4JpqxgO4ud8oLbAZrAsvpesOQEy+qUYMpqPQJjjAl11iMwxpgQZ4XgBETkFRHZLyJrTuK+g0VktYhsEZFpIiI11v1cRDaKyFoReSxQs4rIQyKSJyIrvZdLAjFnjfX3iYiKiE8m2nfoNX1ERFZ5X89/i0jHAM76vyKywZv3AxFpFcBZx3v/n6pF5JS2z59Kvnoe7xYR2ey93FJjeYPvZ79p6nCsULsAZwODgDUncd+lwAhAgM+Ai73LzwW+AKK9t1MCOOtDwH2B/pp613UG5uI5ziQpULMCCTXa3A08F8BZLwQivNf/Cvw1gLOeDpwGfA1kuJHP+9zdai1rA2zz/mztvd76RO9nf16sR3ACqjofKKq5TETSReRfIpIlIt+ISO/a9xORDnj+4Rep5y8+E7jCu/oO4FFVLfc+x/4AzupzDuZ8Argf8NmOLyeyqmpxjaYtfZXXoaz/VtVKb9PFgE/ObO9Q1vWqutHNfPW4CPhcVYtU9QDwOTDW3/93DbFCcHJeAH6uqoOB+4Bn6mjTCcitcTvXuwygF3CWiCwRkXkiMiSAswL8zLtp4BURaR2IOUVkHJCnqtkO5avplF9TEfmLiOQANwAPBnLWGibg+dbqFF9mdUJj8tWlE5BT4/bxzG7+Lt9j5yxuIhGJA0YC79XYnBddV9M6lh3/5heBp4s4HBgCvCsiad5vBYGW9VngEe/tR4C/4flACJicIhIL/A7PZgxH+eg1RVV/B/xORH4D/Az4o4+j+iyr97F+B1QCb/gyY43H91lWJzSUT0RuA37hXdYD+FREjgHbVfVK6s/syu9SFysETRcGHFTVATUXikg4kOW9OQfPB2jNbnQqsNt7PReY7f3gXyoi1XjmJskPtKyquq/G/V4EPvZxRl/kTAe6A9nef9JUYLmIDFXVvQGWtbY3gU9woBDgo6zenZuXAuf7+suKr7M6qM58AKo6HZgOICJfA7eq6o4aTXKBc2rcTsWzLyEXd36XH3Jjx0SwXYBu1NhpBCwExnuvC9C/nvstw/Ot//iOoEu8y6cCD3uv98LTbZQAzdqhRptfAW8HYs5abXbgo53FDr2mPWu0+TnwfgBnHQusA5J9ldHp9wA+2Fl8svmof2fxdjxbAVp7r7dp7PvZHxe/P2GwXYC3gD1ABZ4KPhHPt89/Adnef5IH67lvBrAG2Ao8zf8dwBcFvO5dtxw4L4CzvgasBlbh+UbWIRBz1mqzA9+NGnLiNZ3lXb4KzxwwnQI46xY8X1RWei++GuHkRNYrvY9VDuwD5vo7H3UUAu/yCd7XcgtwW1Pez/642JHFxhgT4mzUkDHGhDgrBMYYE+KsEBhjTIizQmCMMSHOCoExxoQ4KwSmWRCRw35+voU+epxzROSQiKzwzvL5/xpxnytEpI8vnt8YsEJgTJ1EpMGj7lV1pA+f7htVHQgMBC4VkVEnaH8FYIXA+IxNMWGaLRFJB/4BJANHgMmqukFELgN+j+fAvkLgBlXdJyIPAR3xHFFaICKbgC5Amvfnk6o6zfvYh1U1TkTOwTNVdwHQF890CDeqqorn3A2Pe9ctB9JU9dL68qrqURFZyf9NpDcZmOLNuQW4CRgAjAPGiMjvgau9d//B73kKL50JMdYjMM1ZfbNFLgCGe7+Fv41n6urjBgOXq+r13tu98UwjPBT4o4hE1vE8A4Ff4vmWngaMEpEY4Hk888uPxvMh3SDvzK49gfneRbNVdYiq9gfWAxNVdSGeI7z/S1UHqOrWBn5PYxrFegSmWTrBbJapwDve+eCj8Mz9ctwcVT1a4/Yn6jlvRLmI7Afa8f2pgwGWqmqu93lX4ulRHAa2qerxx34Lz7f7upwlIqvwnFTlUf2/ifL6isifgVZAHJ6T7jTl9zSmUawQmOaq3tkigaeAx1V1To1NO8eV1mpbXuN6FXX/z9TVpimnHPxGVS8VkV7AAhH5QFVXAq8CV6hqtojcyvdnsDyuod/TmEaxTUOmWVLPWcC2i8h4APHo712dCOR5r99S1/19YAOQJiLdvLd/eqI7qOom4H+AB7yL4oE93s1RN9RoWuJdd6Lf05hGsUJgmotYEcmtcbkHz4fnRBHJBtYCl3vbPoRnU8o3eHbk+px389KdwL9EZAGe2TAPNeKuzwFni0h34A/AEjynNqy58/dt4L+8Q07Tqf/3NKZRbPZRYxwiInGqelg8G+//AWxW1SfczmVMbdYjMMY5k707j9fi2Rz1vMt5jKmT9QiMMSbEWY/AGGNCnBUCY4wJcVYIjDEmxFkhMMaYEGeFwBhjQpwVAmOMCXH/H/COGcUL3ihzAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr = model.lr_find()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training the model is an iterative process. We can train the model using its `fit()` method till the validation loss (or error rate) continues to go down with each training pass also known as epoch. This is indicative of the model learning the task. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>seq2seq_acc</th>\n",
       "      <th>bleu</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.001017</td>\n",
       "      <td>0.001049</td>\n",
       "      <td>0.999739</td>\n",
       "      <td>0.999311</td>\n",
       "      <td>1:46:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.000797</td>\n",
       "      <td>0.000705</td>\n",
       "      <td>0.999817</td>\n",
       "      <td>0.999521</td>\n",
       "      <td>1:47:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.000314</td>\n",
       "      <td>0.000421</td>\n",
       "      <td>0.999896</td>\n",
       "      <td>0.999720</td>\n",
       "      <td>1:48:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.000213</td>\n",
       "      <td>0.000380</td>\n",
       "      <td>0.999916</td>\n",
       "      <td>0.999773</td>\n",
       "      <td>1:48:06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.fit(epochs=4, lr=lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluate model performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Call `get_model_metrics()` to calculate accuracy [[3]](#References) and bleu [[4]](#References) score on the validation data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'seq2seq_acc': 0.9999, 'bleu': 0.9998}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.get_model_metrics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BLEU : (bilingual evaluation understudy) is an algorithm for evaluating the quality of text which has been machine-translated from one natural language to another. Refer to [this](https://www.aclweb.org/anthology/P02-1040.pdf) paper for details on BLEU score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Validate results\n",
    "\n",
    "Once we have the trained model, we can see the results to see how it performs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "      <th>pred</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>3001.0, ronald reagan cross colorado wb highway, sycamore township ham ohio, ohio, 45236.0, us</td>\n",
       "      <td>3001.0, ronald reagan cross co wb hwy, sycamore township ham oh, oh, 45236.0, us</td>\n",
       "      <td>3001.0, ronald reagan cross co wb hwy, sycamore township ham oh, oh, 45236.0, us</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>702.0, ronald reagan cross colorado wb highway, colerain township ham ohio, ohio, 45251.0, us</td>\n",
       "      <td>702.0, ronald reagan cross co wb hwy, colerain township ham oh, oh, 45251.0, us</td>\n",
       "      <td>702.0, ronald reagan cross co wb hwy, colerain township ham oh, oh, 45251.0, us</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3208.0, ronald reagan cross co eb hwy, sycamore township ham oh, oh, 45242.0, us</td>\n",
       "      <td>3208.0, ronald reagan cross co eb hwy, sycamore township ham oh, oh, 45242.0, us</td>\n",
       "      <td>3208.0, ronald reagan cross co eb hwy, sycamore township ham oh, oh, 45242.0, us</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3102.0, ronald reagan cross colorado eb highway, blue ash ham ohio, ohio, 45236.0, us</td>\n",
       "      <td>3102.0, ronald reagan cross co eb hwy, blue ash ham oh, oh, 45236.0, us</td>\n",
       "      <td>3102.0, ronald reagan cross co eb hwy, blue ash ham oh, oh, 45236.0, us</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7679.0, ginnala connecticut, sycamore township ham ohio, ohio, 45243.0, us</td>\n",
       "      <td>7679.0, ginnala ct, sycamore township ham oh, oh, 45243.0, us</td>\n",
       "      <td>7679.0, ginnala ct, sycamore township ham oh, oh, 45243.0, us</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.show_results()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing model metrics...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "PosixPath('models/add_standardization_4E_bleu_99')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.save('add_standardization_4E_bleu_99')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The trained model can be used to translate new text documents using the `predict()` method. This method accepts a string or a list of strings to translate new documents/text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "txt=['12160, eagle scout connecticut, sycamore township ham ohio, ohio, 45249, us',\n",
    "     '2808, ronald reagan cross colorado wb highway, reading ham ohio, ohio, 45215, us']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='1' class='' max='1', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100.00% [1/1 00:00<00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[('12160, eagle scout connecticut, sycamore township ham ohio, ohio, 45249, us',\n",
       "  '12160, eagle scout ct, sycamore township ham oh, oh, 45249, us'),\n",
       " ('2808, ronald reagan cross colorado wb highway, reading ham ohio, ohio, 45215, us',\n",
       "  '2808, ronald reagan cross co wb hwy, reading ham oh, oh, 45215, us')]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict(txt, max_length=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[1] [The Illustrated Transformer](http://jalammar.github.io/illustrated-transformer/)\n",
    "\n",
    "[2] [Learning Rate](https://en.wikipedia.org/wiki/Learning_rate)\n",
    "\n",
    "[3] [Accuracy](https://en.wikipedia.org/wiki/Accuracy_and_precision)\n",
    "\n",
    "[4] [Bleu score](https://en.wikipedia.org/wiki/BLEU)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.11"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
